Methods and Apparatus for Recognition of Start and/or Stop Portions of a Gesture Using an Auxiliary Sensor

ABSTRACT

Described are apparatus and methods for reconstructing a gesture by aggregating various data from various sensors, including data for recognition of start and/or stop portions of the gesture using an auxiliary sensor, such as a capacitive touch sensor or a MEMS sensor, 
     In a specific embodiment, power savings features are included to preserve the energy stored on the battery of a sensing device, which power savings features are enhanced using an auxiliary sensor, such as a capacitive touch sensor or a MEMS sensor.

RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/924,682 filed Jan. 7, 2014.

FIELD OF THE ART

This disclosure relates to using the Human body as an Input mechanism,and, in particular, recognition of start and/or stop portions of agesture using an auxiliary sensor.

BACKGROUND

Many conventional gestural systems attempt to detect gestures thatresemble characters or words. Such conventional gestural systems,however, offer very poor recognition rates.

SUMMARY

Described are apparatus and methods for reconstructing a gesture byaggregating various data from various sensors, including data forrecognition of start and/or stop portions of the gesture using anauxiliary sensor, such as a capacitive touch sensor or a MEMS sensor.

In a specific embodiment, power savings features are included topreserve the energy stored on the battery of a sensing device, whichpower savings features are enhanced using an auxiliary sensor, such as acapacitive touch sensor or a MEMS sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1(A) illustrates the skeletal rendering of the human with variousnodes, and the usage of many different sensors according to theembodiments.

FIG. 1(B)1 illustrates a system diagram according to an embodiment.

FIG. 1(B)2 illustrates a system diagram according to anotherembodiments.

FIG. 1(B)3 illustrates system diagram according to a further embodiment.

FIG. 2 illustrates that the system allows for the sensor 3 to be usedfor one gesture one pointing to a light (1) as shown in FIG. 2, andanother gesture when pointing at the computer (2) as shown,

FIGS. 3, 4, and 5 show embodiments for micro-gesture recognitionaccording to the embodiments,

FIG. 6 shows an illustration of micro-gestures detected within asubspace that has its own relative coordinate system.

FIG. 7 illustrates a 3D exterior view of a single ring sensor.

FIG. 8 illustrates a more detailed view of the ring sensor of FIG. 7.

FIG. 9 illustrates a computer sensor & receiver according to theembodiments.

FIG. 10 illustrates a flow chart of operation using the capacitive touchsensor and low power modes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Various devices such as computers, televisions, electronic devices andportable handheld devices can be controlled by input devices such as acomputer mouse or keyboard. Various sensors such as accelerometers,gyroscopes, compasses and cameras can be collectively used (all from asubstantially single point such as if disposed on a single ring; or frommultiple different locations) to estimate and/or derive a gesture thatis intended to have some significant meaning. These sensors dynamicallyprovide data for varying periods of time when located in the associatedspace for sensing, and preferably stop or go into a low power mode whennot in the associated space. When sensor data is unavailable, variouscalculations may be employed to reconstruct the skeletal structurewithout all the sensor data.

Various poses and gestures of the human skeleton over a period of timecan be aggregated to derive information that is interpreted (either atthe sensor or at the device) and communicated over wireless channelssuch as WiFi, Bluetooth or Infrared to control various devices such ascomputers, televisions, portable devices and other electronic devices,as described further herein and in the previously filed U.S. patentapplication Ser. No. 14/487,039 filed Sep. 14, 2014, which claimspriority to U.S. Provisional Patent Application 61/877,933 filed Sep.13, 2013, and entitled “Methods and Apparatus for using the Human Bodyas an Input Device”, which are explicitly incorporated herein byreference.

Described are apparatus and methods for reconstructing a gesture byaggregating various data from various sensors, including data forrecognition of start and/or stop portions of the gesture using anauxiliary sensor, such as a capacitive touch sensor or a MEMS sensor.

In a preferred embodiment, MEMS sensors, and preferably a plurality ofthem within a substantially single location such as on a ring, or in ahead mounted device, or in a capsule either directly mounted on the bodyor enclosed in a garment or clothing, or some other wearable form factorare used, in combination with a capacitive touch sensor or a tactileswitch or sensors used for recognition of start and/or stop portions ofthe gesture. MEMS sensors provide the advantage of not requiring aseparate detector compared to conventional camera based depth sensorsand don't have to be in the very restricted viewing area of aconventional depth camera. A plurality of MEMS sensors can be used toobtain further information than would be possible with a single suchsensor, as described herein. When further used in combination withaccelerometers, gyroscopes, compasses, the data from the various sensorscan be fused and interpreted to allow for sensing of micro-gestures, asdescribed herein.

Such a single sensing device having multiple sensors can be integratedinto everyday objects such as clothing, jewelry and wearable deviceslike fitness monitors, virtual reality headsets, or augmented realityglasses in order to use of the human body as a real-time input devicethat can interact with a machine in its surroundings.

Processing of all the data generated to accurately detect the pose ofthe human body in real-time includes engineering desiderata of eventstream interpretation and device power management, as well as usage ofalgorithms such as Kalman filtering, complementary filters and otherconventional algorithms used to fuse the sensor data into coherent poseestimates. The filtering algorithms used are based on the locality ofthe sensor and factor in the human anatomy and the joint angles of thebones the sensors are tracking. The fused data is then processed toextract micro-gestures—small movements in the human body which couldsignal an intent, as described herein.

Gestures such as waving your arm from one side to another ormicro-gestures such as swiping your index finger from one side toanother are mapped to functions, such as changing channels on a TV oradvancing the song being played. More complex gestures, such asinteracting with the User Interface of a tablet computer are alsopossible using micro-gestural primitives to generate a more complexmacro intent that machines in the environment can understand. All ofthese gestures, however, must have start points and stop points, whichneed to be detected in some manner.

Thus an aspect of the system includes assembling a movement sequence(aka gesture) that could be used to indicate a command, for example,which has a start point and a stop point. Each gesture can also take ona different meaning depending on which device it is communicating with.Thus, pointing to a Television and moving your hand from one directionto another can imply changing the channel while a similar such gesturecould imply changing the light intensity when done pointing to a lightbulb, with each of the Television and the light bulb being separatesubspaces that are detected as such by an overall detector, for example.

Efficient power management strategy is also provided, such that thesensor device doesn't require a power on or power off switch. Thisinvolves determining the current state of gestural detection and furtherincludes the ability to turn off components such as the gesturaldetection unit, or various sensors to save power, and in particularusing a capacitive touch sensor or a tactile switch or a specificgesture or any combination of the three as described hereinafter toaccomplish certain of these power savings.

It is noted that the single sensing device is a battery-operated device,yet it does not necessarily have a power button. It does, however, havecapacitive touchpads and tactile switches as described, which can beprogrammed to activate and/or de-activate the single sensing device,thereby ensuring that the device is in use only when the user intendsfor it to be and keeping it energy-efficient.

As described further herein, an auxiliary sensor, such as a capacitivetouchpad or a tactile switch on the wearable input platform, alsoreferred to as single sensing device or ring herein, upon receiving aspecific input (i.e. tactile, capacitive touch, gesture or combinationthereof) from the user, enables the communication and connectivitychannels on the platform and signals to the gesture acquisition engineto start acquiring gesture data and manipulating such gesture data tointeract with a specific application device. Similarly, the same ordifferent touch input, when applied to the touchpad, can disable (orsend into an idle state) the communication and connectivity channels onthe platform to signify an end to the interaction with the device,thereby stopping the gesture acquisition engine from continuing toacquire data.

This capacitive touch sensor and tactile switch feature takes theuncertainty out of the gesture acquisition engine, whereby it is nottrying to interpret a random gesture, unless expressly instructed to doso, via the touch input imparted to the capacitive touchpad or tactilebutton. Additionally, the capacitive touch sensor feature ensures thatthe single sensing device is energy efficient and active only as neededfor the duration of the interaction. Similarly, the gesture acquisitionengine is not “always-on” and in use only when needed, therebyconserving energy.

The specific input imparted to the touchpad can vary, depending onadditional touch gestures that could be programmed, depending on the usecase. If the wearable input platform is purely intended for gesturecontrol, then any kind of input would suffice for the purpose ofstarting and stopping the gesture acquisition. However, there may beinstances, where the wearable input platform could be used to control amusic player on a smartphone or some similar device, thereby requiringmore than one type of input. In such an instance, a long contact withthe touchpad could signal the start of the device control using theinput platform. Further, a short contact with the touchpad couldindicate pausing the track, a swipe to the left could mean going to theprevious track, etc. The preceding touch inputs are meant to be anexample of what is possible for a given use case. Hence, in such ascenario, it is important that the start/stop gesture detection inputsare sufficiently distinguished from other device operation touch inputs.

These various aspects are shown in the diagrams attached. FIG. 1(A)illustrates the skeletal rendering of the human with various nodes, andthe usage of many different sensors: one on the glasses (1), another onthe belt (2), a third of a number of different sensors for fingers (3),one for the wrist (4) and one on an ankle bracelet or attached to thebottom of the pants worn (5). FIGS. 1(B)(1-3) shows a similar space andrendering, and points out specific sub-spaces associated with differentobjects; each of which can have their own relative coordinate system ifneeded. As shown, FIG. 1(B)1 illustrates a system diagram with a laptopas a third controllable device, which laptop includes an interactionplane and is labeled as Computer Sensor & Receiver to illustrate that itcan operate the software needed to fuse different sensor data together,as described elsewhere herein. FIG. 1(B)2 illustrates a system diagramwith a laptop as well, but this laptop shown only as having aninteraction plane, and operate upon a distributed system (such as withcloud processing). FIG. 1(B)3 illustrates an even simpler, which doesnot include the laptop at all within it. As is apparent, many differentcombinations are possible and within the contemplated scope herein.

As described above, the system allows for the sensor 3 to be used forone gesture one pointing to a light (1) as shown in FIG. 2, and anothergesture when pointing at the computer (2) as shown.

FIGS. 3, 4, and 5 show embodiments for micro-gesture recognition thatinclude usage of 1, 2 and 3 finger rings, respectively, as shown. Otherconfigurations are possible and within the intended scope herein.

FIG. 6 shows an illustration of micro-gestures that are detected withina subspace around a computer, which sub-space can have its own relativecoordinate system, rather than being based upon absolute coordinates. Inaddition to the MEMS sensor data in each ring, radio strength can alsobe used to detect distance from a relative reference point, such as thescreen of the computer. Additionally, the relative coordinate system canbe based on the part of the body to which the single sensing device isattached, with a ring on a finger having as a relative coordinate systemthe portion of the arm from the elbow to the wrist as one axis.

FIG. 7 illustrates a 3D exterior view of a single ring sensor, and FIG.8 illustrates that ring sensor in a more detailed view, with thesignificant electronic components identified, and which are connectedtogether electrically as a system using a processor, memory, software asdescribed herein, including other conventional components, forcontrolling the same. The processor controls the different sensors onthe ring device and is in charge of detecting activity in the varioussensors, fusing the data in them and sending such data (preferablyfused, but in other embodiments not) to other aggregators for furtherprocessing. While shown as a ring sensor, this combination of elementscan also be used for the other sensors shown in FIG. 1—though othercombinations can also be used. Note that while only a single capacitivetouch sensor is shown, that multiple capacitive touch sensors can beincluded and with tactile switches

FIG. 9 illustrates a Computer Sensor & Receiver as shown in FIG. 1(B1).As illustrated in FIG. 9, included are a processor, memory and displaythat are used as is conventionally known. The processor controls thedifferent sensors on the various devices and can fuse the data fromdisparate devices that has been aggregated previously or not, and sendsuch data (preferably fused, but in other embodiments not) to otheraggregators for further processing as well as send control signals basedon the what has been detected to control devices such as the light ortelevision as shown in FIG. 1. I/O devices as known are also included,as well as what is labeled a Gesture Input/Output Device and anAggregator coupled thereto (which Aggregator may be part of the ComputerSensor and Receiver or could be located elsewhere, such as on a wristsensor as described above). The Aggregator can be implemented inhardware or software to process the various streams of data beingreceived from the various sensors. The Aggregator factors in location ofthe sensor (e.g: on the finger or wrist etc.) and calculates what datais relevant from this sensor. This is then passed on to the GestureInput/Output Device (which could also reside across a wireless link) tocontrol various computing devices.

The device that could be worn on the ring could possess a CapacitiveTouch surface or a tactile switch on the exterior of the device(preferably the entire exterior surface or an entire portion of anexterior surface associated with a single Capacitive Touch Sensor ormultiple touch-sensitive areas of varying lengths and sizes) and aCapacitive Touch Sensor enclosed in the inside of the device.

The device can also possess a haptic actuator and associated circuitryto be able to provide a haptic feedback based on user engagement with acomputing device. The device can also support various forms of wirelessnetworking such as NFC, Bluetooth and/or WiFi to be able to interactwith various other devices in its surroundings.

Multiple sensors can interact with each other providing a stream ofindividually sensed data. For example a sensor worn on the ring cancommunicate with a wrist worn device or a smartphone in the pocket. Thisdata could then be aggregated on the smartphone or wrist worn devicefactoring in the human anatomy. This aggregation may factor in range ofmotion of the human skeletal joints, possible limitations in the speedhuman bones could move relative to each other, and the like. Thesefactors, when processed along with other factors such as compassreadings, accelerometer and gyroscope data, can produce very accuraterecognition of gestures that can be used to interact with variouscomputing devices nearby.

FIG. 10 illustrates a flowchart of the preferred operation using thecapacitive touch sensor and low power modes, which is implemented inapplication software loaded onto the memory and executed by theprocessor, in conjunction with the gesture input/output device,aggregator, and sensors. For understanding, operation of a singlesensing device is explained, but it will readily be appreciated that thesame operations are used for multiple sensing devices, with then one ofthe sensing devices and/or control devices being the master device.

In operation, step 1010 of FIG. 10 shows the single sensor device beingin the “on” state and charged sufficiently for operation. If notcharged, then a separate charging station (not shown) can be used tocharge the device. After step 1010, step 1012 follows, with entry into alow power mode. In this low power mode, the minimum operations areperformed, and as many of the MEMS sensors and the like are put into asleep state in order to preserve power, with the auxiliary sensor, suchas the capacitive touch sensor, being periodically awaken and scanned tosee if an event has occurred, in steps 1014. Further, other start events(such as tactile or gestural input) can be programmed, and this is shownas step 1016. In a preferred embodiment, the low power mode has atactile only input to wake up from deep sleep, and all other sensors areoff, as well as wireless transmission/reception. In both the medium andlow power modes, wireless transmission/reception is preferably off.

If in either of steps 1014 or 1016 a start signal is detected, thensteps 1018 follows, with the single sensor device entering the regularpower mode, such that full functionality is possible, though even withinfull mode power savings procedures can be put in place to conservepower.

One step as shown in the regular power mode is indicated as step 1020 inwhich gesture and other data are detected, until a stop signal isdetected. Other full functionality steps can also occur, such asProcessing/transforming the gestures and other sensor data such asacceleration and orientation; transmitting the processed data over awireless medium to enable interaction with the smart device (TV, smartphone, tablet, etc.)

Steps 1022, 1024 and 1026 all follow, which are each detecting theexistence of the end of the gesture. Usage of the capacitive touchsensor to detect a specific stop gesture is shown in step 1022, whereasstep 1024 shows that an end of gesture can be detected based upon thegesture data (based on a pre-programmed, unique gesture). Step 1026indicates that a time limit or other stop trigger (such as a tactileswitch) can also be used to generate the stop signal at the end of agesture.

Upon detection of a stop signal in any of steps 1022, 1024 and 1026,step 1028 follows, and a medium power mode is preferably entered into,in which case the, for example, no further gesture data collection isperformed, the MEMS sensors are turned off, and processing of thegesture data collected already is finished using time-keeping, so as tothen perform operations in accordance with such processed gesture data.Other functions that may still occur in a medium power mode, that wouldpreferably not occur in a low power mode, are keeping all the sensorssensing (in standby) and waiting for some combination or one oftouch/gesture/tactile input for quick startup.

Following step 1028 is a step 1030, in which a preferably programmeddetermination of whether to then enter into the low power mode 1012, theregular power mode 1018, or stay in the medium power mode 1028.

Although the present inventions are described with respect to certainpreferred embodiments, modifications thereto will be apparent to thoseskilled in the art.

1. An apparatus capable of interacting with at least one controllabledevice based upon a pose of at least a portion of a human body, theapparatus comprising: one or more sensors that are sized for wearing onthe human body, each of the one or more sensors emitting sensor data; anauxiliary sensor sized for wearing on the human body that receives afirst specific input based on one of a tactile switch and capacitivetouch input and generates a gesture start signal; and a detection unitthat operates upon the sensor data to determine the pose of at least theportion of the human body and is capable of interacting with the atleast one controllable device, the detection unit including: a memorythat stores at least one or more characteristics of human anatomy thatare associated with the human body using at least a partial skeletalrendering of a human; and a detection processor, automatically operatingunder software control, that inputs, aggregates and fuses the sensordata from each of the one or more sensors using the at least one or morecharacteristics of human anatomy stored in the memory to determine thepose of at least the portion of the human body based upon a locality ofsaid one or more sensors, wherein the detection processor begins toinput, aggregate and fuse the sensor data upon receipt of the gesturestart signal and ceases to input the sensor data upon receipt of agesture stop signal; wherein at least some of the one or more sensors,the auxiliary sensor, and the detection unit are packaged in anintegrated mechanical assembly.
 2. The apparatus according to claim 1wherein the auxiliary sensor receives a second specific input based onone of the tactile switch and the capacitive touch input and generatesthe gesture stop signal.
 3. The apparatus according to claims 2 whereinthe first specific input and the second specific input are the same. 4.The apparatus according to claims 2 wherein the first specific input andthe second specific input are different.
 5. The apparatus according toclaim 1 further including a timer that is used to create the gesturestop signal a predetermined period of time after generation of thegesture start signal.
 6. The apparatus according to claim 1 wherein thefirst specific input is further used to change a power mode associatedwith the detection processor.
 7. The apparatus according to claim 1wherein the first specific input changes the detection processor into aregular power mode and the gesture stop signal causes the one or moresensors to be turned off.
 8. The apparatus according to claim 1, whereinthe apparatus is further configured to interact with a first device anda second device, and wherein an orientation of the apparatus relative tothe first device causes the pose to be used to signal the first device,and wherein orientation of the apparatus relative to the second devicecauses the pose to be used to signal the second device.
 9. The apparatusaccording to claim 1 wherein the detection processor receives furthersensor data that represents a stop time, and the gesture stop signal isgenerated therefrom.
 10. The apparatus according to claim 1 wherein thedetection processor receives further sensor data that represents a starttime, and the gesture start signal is also generated therefrom.
 11. Anmethod for interacting with at least one controllable device based upona pose of at least a portion of a human body, the method comprising:sensing, using one or more sensors that are sized for wearing on thehuman body, sensor data from each of the one or more sensors; sensing,using an auxiliary sensor sized for wearing on the human body a firstspecific input based on one of a tactile switch and capacitive touchinput and generating a gesture start signal; and determining the pose ofat least the portion of the human body based upon the sensor data, underprocessor and software control, the step of determining operating to:associate at least one or more characteristics of human anatomy with thehuman body using at least a partial skeletal rendering of a human; andautomatically determine, under the processor and software control thepose of at least the portion of the human body based upon a locality ofsaid one or more sensors and the input from the auxiliary sensor, thestep of automatically determining including inputting, aggregating andfusing the sensor data from each of the one or more sensors using the atleast one or more characteristics of human anatomy to determine thepose, wherein the step of automatically determining begins to input,aggregate and fuse the sensor data upon receipt of the gesture startsignal and ceases to input the sensor data upon receipt of a gesturestop signal, and wherein the at least one or more characteristics ofhuman anatomy that are associated with the human body that are stored inthe memory include at least one of (a) a range of motion of humanskeletal joints and (b) limitations in the speed human bones can moverelative to each other; and wherein at least some of the one or moresensors, the auxiliary sensor, he processor and the software arepackaged in an integrated detection unit mechanical assembly.
 12. Themethod according to claim 11 further including the step of generatingthe gesture stop signal from the auxiliary sensor sized for wearing onthe human body, the gesture stop signal obtained based upon a secondspecific input based on one of the tactile switch and the capacitivetouch input.
 13. The method according to claim 11 wherein the auxiliarysensor receives a second specific input based on one of the tactileswitch and the capacitive touch input and generates the gesture stopsignal.
 14. The method according to claims 13 wherein the first specificinput and the second specific input are the same.
 15. The methodaccording to claims 13 wherein the first specific input and the secondspecific input are different.
 16. The method according to claim 11wherein a timer is used to create the gesture stop signal apredetermined period of time after generation of the gesture startsignal.
 17. The method according to claim 11 wherein the first specificinput is further used to change a power mode associated with theprocessor.
 18. The method according to claim 11 wherein the firstspecific input changes the processor into a regular power mode and thegesture stop signal causes the one or more sensors to be turned off. 19.The method according to claim 11, wherein the integrated detection unitmechanical assembly interacts with a first device and a second device,and wherein an orientation of the an integrated detection unitmechanical assembly relative to the first device causes the pose to beused to signal the first device, and wherein orientation of theapparatus relative to the second device causes the pose to be used tosignal the second device.
 20. The method according to claim 11 whereinthe processor receives further sensor data that represents a stop time,and the gesture stop signal is generated therefrom.
 21. The methodaccording to claim 11 wherein the processor receives further sensor datathat represents a start time, and the gesture start signal is alsogenerated therefrom.